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PREFllCE 



This document is the result of contributions from a National Work 
Group organized under the auspices of the Office of Refugee 
Resettlement of the Family Support Administration. The National 
MELT nork Group met once in Washington, D.C. and once in Colorado 
with snaaller task forces meeting on two separate occasions. It 
was a tremendous challenge to write a document which woulc; be 
useful to a wide range of audiences and be applicable and 
appropriate across states, whether urban or rural, large or small, 
and with ELT programs at all levels of sophistication. 

Members of the MELT Work Group were chosen for their experience and 
leadership with ELT programs and the MELT Initiative, as well as 
for their knowledge and involvement in issues facing policy-makers 
regarding ELT programs. The group was composed of educators 
involved with English language training. Refugee 'State 
Coordinators, Representatives from the FSA Regional Offices, and 
the Central Office of Refugee Resettlement. Members of the 
National Work Group included: 

Myma Ami AdkiiiS, President, Spring Institute for 

International Studies, Denver, Colorado 
Walter Barnes, California State Refugee Coordinator 
John Crossman, Progrzua Manager/Work Program and Refugee 

Resettlement, FSA, Region X, Seattle, Washington 
Allene Grognet, Associate Director, Center for Applied 

Linguistics, Washington, D. C. 
Russell Jalbert, formerly ORR/FSA Region I, Boston, 

Massachusetts 

Autumn Keltner, formerly ABE/ESL Coordinator, San Diego 

Community College District, Continuing Education Centers, 

currently Educational Consultant 
InasuB Mansoor, REEP, Arlington Education and Employment 

Program, Arlington Public Schools, Virginia 
Linda Mrowicki, Director of Project Work English, Northwest 

Educational Cooperative, Des Plaines, Illinois 
Diane Pecoraro, Minnesota Department of Education, Adult 

Refugee Education Program Specialist 
Lee Russell, Texas State Refugee Coordinator 
Barbara Sample, Director of Educational Services, Spring 

Institute for International studies, Denver, Colorado 
K. Lynn Savage, Director, ESL Teacher Institute, Association 

of California School Administratio and Vocational ESL 

Resource Instructor, San Francisco Community College 
Pamela Seubert, Program Director, Refugee Social Services 

Program, Jewish Federation of Metropolitan Chicago 
Edwin Silverman, Manager, Naturalization Service Section, 

Bureau of Program Services, Illinois Dept. of Public Aid 
Carmel Thompson, Program Analyst, ORR/FSA Central Office, 

Washington, D. C. 

i 
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The Spring Institute for International studies, through its MELT 
Technical Assistants Contract for Region VIII, facilitated the 
meetings of the Wt rk Group and produced the final version of this 
document. The Institute owes much to the hard work, time, and 
substantive input contributed by every member of the work Group. 
The Institute also gratefully acknowledges the leadership and 
active participation in the work group by Mr. Vo Van Ha, Project 
Officer from Region VIII, and Carmel Thompson, from the National 
Office of ORR. Much credit is also due to Kathy Do for che 
original design of the project » and for her contributions in the 
start-up phase and her leadership throughout the earlier phases of 
the MELT and VELT projects. 

The five chapters are distilled from the experience of a wide range 
of leaders in language teaching and refugee services throughout the 
United states. Chapter I deals with the decisions to be made 
regarding the scope and design of an English language training 
program for a given community, taking into account the clieflts to 
be served, the needs of the service locale, and ways of 
prioritizing services. caiapter 11 suggests criteria to be used 
when measuring the quality and effectiveness of a program, and 
Chapter m describes procedures for monitoring and evaluating a 
program in relation to those criteria, chapter IV discusses ways 
to coordinate the ELT program with other elements of the service 
delivery system, so as to ensure efficiency and economy, through 
communication among teachers, case managers, job developers, and 
employers. The document concludes with a consideration of the 
costs, as Chapter V offers help to policymakers responsible for 
efficient allocation of resources while purchasing needed services. 
In each chapter, selection of content has been guided by the 
question: how can the experience of the work Group members best 
serve those entrusted with designing, guiding and financing ELT 
progrzuDs for refugees? 

Since the development of the MELT products, many statewide systems 
and local programs have adapted them or developed their own 
curriculum, tests, and definitions of performance to meet their 
clients' needs and better to address the unique features of their 
competency-based programming. 
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CHAPTER I 



IDBMTIFYIMG TEE SCOPE AND DESIGN Oi? LANGUAGE SERVICES 



UVSKVIEW 

The design of an English language Training (ELT) program should be 
based on the characteristics and needs of the potential clients. 
This chapter addresses the steps involved in identifying the scope 
and design of effective ELT prograuns to correspond most closely 
witA the clients and the language skills they need for available 
jobs. 



KEcowamaaiaaa 

1- A survey M hould be eondugfd to identify i:ha clients to be 
served and to aaaeas iihe ir needs. 

The types of client information appropriately collected in a survey 
inciade: 

. Demographic data 

. Length of time in the United States 

. Language background 

. Level of English proficiency 

. Public assistance status 

. Employment history and current status 

To complement the client survey, a survey of major industries and 
primary employers in the service locale will help to assess: 

. Types of available jobs 

Language demands of the workplace 

The survey instrument presented in the MELT Technical Assistance 
Package may be used or adapted to the local situation. 

2- T»IM tt«ed to describe laaan^oa yy ftfieienev should ba elaar. 
im4frrtOT4rt1t to ^ <iwr«l audience, and eonaiat ant with an 

•gffWtf^ t^OMra. Thm MBLT Student Pa rformanea l^ala 

CegLsl . or ecraallY deaoriotive tarma wh ich ara eonaiatant with 
the flPLa. ara raooMiandad. 

The (SPLs) found in the MELT Document are general descriptions of 
adult students* language ability set at a range of levels. They 
provide a common standard for refugee ELT level descriptions and 
can be used by programs nationwide. They facilitate understanding 
of language abilities within a single ELT program as well as among 
ESL programs. The SPL Document provides a basis for communication 

1 
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with providers of other services (e.g., resettlement, job 
plac«Bent) and with funding sources. Finally, they afford a basis 
for identifying the relationship between overseas and domestic 
refugee ELT progreun levels. 

3* h Tmaam of BLT aervieaa aheuld ba n rovidad for a varlaty of 

cli<nt goals and eoMnmitv nmmda. 9ratjj^»^f ^^ ompleiiiantiarv to 

gar* snch >s. Vocational Enaliah i^onaaa Tra ining fVBLT^ and 

AWt BMig BdaoatiPB (MB). ah ould ba Idantlfiad and 

accasaad. 

The program design should reflect outcomes that are specifically 
associated with the program's goals. Thus, where self-sufficiency 
is the goal in a program targeted to the unemployed or 
underemployed head-of -household, appropriate outcomes may include 
employment at a designated salary level or advancement within a 
specific period of tine to that level. For homemakeis and for non- 
literate older persons, the goal may be the ability co function in 
the community. 

VELT, ABE and other programs complementary to ELT should be 
identified so that ELT programs are designed to accommodate the 
needs (e.g., hours and level of instruction) of a target population 
who may be simultaneously using these community resources. 

4* Prioritiea for aarvioaa to ba funded or elianta to ba aarvad 
ShOWld be establiahad baaed on Fad aral guidalinea and Stata 
ebiaettvaa. 

Priorities are set In part by determining whether programs will 
serve those clients with the lowest level of English proficiency, 
those most likely to be employed in jobs available ir the 
community, or those meeting some other identified standard. A 
factor which should also be considered is the extent to which 
services are availsJale in the community. 

A related method of establishing service priority is to consider 
the priorities specified in the state plan and apply them to 
English language services. Groups to be targeted for service 
might, without reference to priority, include: 

a. Employable adults receiving cash/m«dical 

assistance amd resident less than 24 months. 

b. Employable adults resident 0-6 months and in danger of 

seeking public cash/medical assistance. 

c. Secondary wage earners receiving cash/medical assistance. 

d. Secondary wage earners resident 0-6 months and in danger 

of seeking public cash/medical assistance. 



e. Time-expired employable adults on cash/medical 

assistance. 

f. Sp«*cial needs populations, such a:3, homebound women, 

elderly, and youth. 

g. All others who have not become naturalized citizens. 

An altomative way of prioritizing services is to set a limit on 
the number of hours or length of time in the program. This method 
may ensure that some opportunity for ELT instruction will be 
available to the entire target population for whom lack of English 
is a barrier. 

A variation on this method of prioritizing clients is to identify 
the proficiency level at which clients may be enrolled in ELT or 
the level at which clients must also be employed to be eligible for 
ELT. - 

If a limit is to be set on the length of time a client may spend 
in an ELT program, a review of the number of contact hours required 
on average to reach a certain proficiency level is helpful. The 
MELT Resource Package presents the following guidelines for 
estimating the ranges of contact hours required for clients to move 
from one (SPL) to another. The ranges of hours listed below 
represent the experience of 19 MELT demonstration sites. The table 
is intended to be used as a guideline for ELT programs that choose 
to adopt or adapt MELT products. Th^j table illustrates that 
individuals with virtually no English sicills may require over 1,000 
contact hours to reach Level VI English language proficiency. 



Range of Contact Hours Required for Level Advancement 



Gain in SPL Related to Listening Number of 
Coapreheasion and Oral ComBunication Contact Houra 

I to II 105 to 235 

II to III 125 to 210 

III to IV 120 to 210 

IV to V 120 to 225 

V to VI 120 to 225 

VI to VII 120 to 225 



Gains in language proficiency depend on conditions related to the 
local program and the individual student. The contact hour ranges 



8 



are also based on the assumption that certain conditions related 
to the teaching/learning environment exist. Interpretation of the 
contact hour ranges must, therefore, include a review of these 
conditions and the flexibility to institute adjustments, if 
necessary, to accommodate differences in conditions. 



SE80USC18 

o Appendix B: student Performance Level (SPL) Document 

o Appendix C: Assessing the Needs of students. Community, and 
Program. 

o MELT Resource Paelcagrft, Appendix 111.5-111.7. Sample "Needs 
Assessments. 

® VELT Reaource Paekaofti Section Two: Vocational English 
Training Program, Key Components, Pp. 2-1 to 2-32. 



CHAPTER II 



lOEMTZFYniG IHDZCATORfl OF PROGRAM QUALITY 

0VBR7ZSW 

This chapter deals with the elements of good ELT progreuns which 
help to ensure that programs meet the needs of the students served, 
are outcome based, and have goals consistent with State priorities. 
In the procurement, monitoring and evaluation stages, the elements 
listed below characterize programs in which the quality of 
instruction is high. 



DB8CSIPTI0H AMD DISCaSSZOV 

1« BPecifie student Qntoomes a re related to identified student 
needs. prooraB goals, and fltate pr iorities. 

An important measure of program effectiveness is the percentage of 
participants who achieve the proposed student outcomes. These 
should be stated in terms of measurable performance, be achievable 
by the target population, and be consistent with the established 
program goals, objectives and State priorities. The following are 
examples of how the proposed student outcomes are related ' to 
student needs and State priorities: 

Example 1; 

State Priority Population: 

Employable adult recipients of cash or medical assistance. 

Student Needs: 

75% of jobs in the local labor market require an ability to 
read written instructions. 

Program Type: 

Vocational esl. 

Proposed student Outcome: 

Clients will gain sufficient literacy skills to enable them 
to read written instructions related to the following jobs in 
the local labor market: (Enumerate available jobs and attach 
samples of written instructions for each) . 

Example 2: 

State Priority Population: 
Older Adults. 

Student Needs: 



to 



To access the health care system and other support services 
by public transportation. 

Progrzun Type: 

Survival English. 

Proposed student Outcome: 

Clients will learn the language skills necessary to eneible 
theu to use public transportation. 

2* OtttCOMM mx% eroressed in the eurrieiilum in fcerms of llfe« 
skills flometiifllea. 

As the outline of the instructional program, the curriculum 
provides the guidelines and frameworlc for student outcomes. Each 
l«vel of each component in a program design should have a separate 
curriculum, (e.g, , Survival English 1, 2; Occupation-Specific 
English 1, 2, 3.) 

If acquiring fxmctional life skills is the intended program goal, 
em appropriate curriculum is one that is competency-based, focusing 
on demonstrated mastery of basic and life skills necessary for 
functioning capably in society. 

In a competency-based curriculum, grammar is not taught in 
isolation, nor is it the rationale for sequencing lessons. Rather, 
the grammatical structures are those needed to demonstrate 
attainment of given competencies. The teacher focuses instruction 
021 the language a student needs (e.g., to apply for a job, access 
the health care system, or report an emergency) and integrates the 
appropriate graiamar skills into the content of the lesson. Where 
a life skills, competency-based curriculum is in place, it is 
recommended that the competencies be correlated with the SPL's. 
If such a curriculum is not in place, it is recommended that the 
Core Curriculum document from the MELT Project be adapted to 
develop a curriculum based on local needs. 

3. Cross-cnltnrel content is ino luded in the cnirrioulum . 

Functioning effectively in the United states requires an 
understanding of Aaerican cultural norms, values and behaviors and 
their relation to one's native culture. An effective ELT program 
includes cultural content, both native and American. 

4* If more thmn one level of instruction is offered, the oroorai^ 
Wt MeMHres to assure accurate pl acement of students into 
aPBronriete Ymvmla. 

Placement measures are administered at the time a student enters 
a program. To place students into appropriate levels of 
instruction, measurement must effectively discriminate individual 
students' abilities by local program level. Refugees' 1-94 cards 

6 
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indicate the level of instruction (A, B, C, D, or E) they have 
completed in the Departner**-. of state Refugee Training Program 
overseas. Ttiis level should be used only as one indicator since 
students were not individually tested on exit nor did they move 
from level to level in the camp program. Additionally, time and 
experience since leaving the processing center may also have 
significantly modified refugees' proficiency levels. Thus, it is 
important to reassess the proficiency level at the time of a 
student's entrance into the program. In the eibsence of another 
equally valid and reliable test, It is recommended that the Basic 
English Skills Test (B.E.S.T.) be used for placement. If a student 
is more proficient them is measureable with the B.E.S.T". (eU^ove 
SPL 7), other standardized tests may be used.* 

BLT programs ocmalder the follo wina eonditiona uhmn 

jtf IBininq the length of time b— ded to move fr om one l^vel 

to the n«rt. 

a. PROGRMC-RBIATBO C0MDITI0H8 

. Intensity of Instruetion ~ if funding peiiBits, programs 
should offer between 10 and 25 hours per week with at 
least three class sessions per week. 

Bntry/Bzit Proeeduxes and Policies — where open- 
entry/open-exit policies exit, it may be necessary to 
extend the range of contact hours or establish 
procedures which accommodate midrterm admissions. In 
programs with definite starting and ending dates, the 
range of contact hours will be more definitive. 

. Class 8ise — A maximum of 15 students per class for 
Levels I and II is recommended, while a maximum of 25 
students per class is suggested for higher levels. 

Class Composition (Students) — Classes should be 
homogeneous in competency level. Multi-level classes may 
require more contact hours to compensate for the ranges 
of levels. 

. Teachers — Teachors should be trained and supervised 
during the implementation of competency-based ELT 
curricula. It is desirable that teachers have an 
academic training in ESL or substantial experience and 
^n-service training in using the methods and curricula 
appropriate for the target population. On-going staff 
development is necessary to operate a competency-based 
ELT progrzun. 

* (See Reviews of English Lanmiage Profici encv Tasts . a TESOL 
publication by Alderson, Krahnke, Stansfield, and Appendix E 
for a brief description of some of the tests.) 

7 
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. Appropriat* curricula and Coursa material — • Because 
there is a ono-to-one relationship between the MELT 
Core Curriculum document and the SPL's, the local 
curriculum must be competency-based with the same or 
similar competencies and relationship to the SPL's as the 
MELT curriculum in order to move students from one 
Student Performance Level to the next within the 
recommended range of contact hours. 

. Testing and Assessment — Pre-detennined testing and 
assessment procedures are necessary. 

. Auxiliary services — Auxiliary services, such as child 
care and transportation, provided by the program or 
through linkages with other agencies, enhance the 
possibilities for regular attendance in ELT programs. 

b. STUDIST-RBLATBO C0MDZTZ0H8 

. Age — Evidence suggests that contact hour raquirements 
increase in direct proportion to increased age. 

. Education — students with less education will typically 
require more contact hours than the stated range. 

. Native Language — students whose native language uses 
a non-Roman alphzOset may require more contact hours. 

Hative Language Literacy — students who do not read or 
write their native language may require more contact 
hours. 



. Use of English Outside class — students who are isolated 
from English speakers may require more contact hours. 



6- Tht PrpgXM PWYi^es A record of prooreaa through or— & id 

g?f?Z"***""^ tests, which are approortate . rel i able, and 
▼*1^*- 

Pre- and post-assessments should be conducted in order to assess 
students' language skills, to identify what need*-, to be taught, to 
place students in the appropriate instructiou,».l levels, and to 
provide data to document progress from the tir.a of entry to the 
time of completion, in many programs the placement measure and 
pre-assessment tost are the same instrument. A test which is 
appropriate for pre- and post-assessment mirrors the objectives of 
the curriculum. A reliable test is one wi.ich is accurate in 
measuring consistently what it purports to measure. Vallditv 
refers to the degree to which a test is relevant and meaningful in 
measuring a particular competency or trait for a particular purpose 
and a particular examinee. ^ ^ ^ 
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In determining language proficiency lev«*l, it is recommended that 
a test be used which is both appropriate to the needs of the target 
population and related to the proficiency le/els. One such test 
is the B.E.S.T. , which is a competency-based test developed 
specifically for the refugee population but w^ich is appropriate 
for any adult ESL population. It is correlated to the SPLs. 

7. Procedures far ■oBlterina afc udent nroareaa on a regular 
basie are la 

Progress is monitored to determine whether a student is ready to 
move on to the next level. Progress is recorded so as to document 
the attainment of competencies. Therefore, tests used for this 
purpose should measu re the skills and competencies included in the 
local curriculum. 

Teacher-made tests are appropriate for this purpose since they can 
be directly related to the content of the course. In a competency- 
based approach, students can demonstrate mastery by performing the 
tasks that have been taught. 

ProaraB outeoaea are Measure d bv use of standard definitions 
of anoceaaful eoMPletion. 

Definitions of coB^letion may include: 

. Movement from one level to another. 

. Attainment of a specific set of competencies. 

. Completion of a certain number of levels. 

. Attainment of a designated test score on a reliable and 

valid instrument. 
. Movement into emother program such as a vocational 

training program, which requires a higher level of 

English, or 
. Getting and keeping a job. 

Materials appropriate to the ta aohina obHectivesand student 
needs are ummA. 

Emplxasis should be placed on what a student can do with English 
rathar than on what he/she knows about English. Thus, 
instritctional materials should focus on life skills competencies. 
A single iext is usually not sufficient to meet the needs of 
students ^.n any one locale. Instead, teachers may draw from a 
variety of resources including the community itself. For example, 
to learn how to use public transportation, local bus schedules and 
maps may be used instead of, or in addition to, a text. 

10- atatf development aativitlea a re provided on a racnilar baaia. 
A Pltn for addressing staff develoiMf nt naeda ia an aaa^ntial 
gomponeat of all nrepoaals^ and prog ram planning. 

9 
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AIJ staff need to acquire information on changing guidelines and 
reqairexnents, new methods and materials, and program evaluation 
results. As staff turnovers occur, new staff must be oriented to 
the goals and objectives of the program and to specific iob 
responsibilities. 

Additional topics to cover in staff development might include: 

. use of materials, including audio visual materials 

. cultural information 

. ways of teaching to the needs of the students and proposed 

studttnt outcomes 

. lesson planning 

. time manageaent 

. monitoring progress 



RBSOURCSS 

o Appendix A: Sample pages from the Basic English Skills Tf ^^ ^i ; are 
included in the Appendix on Testing, section iv of the MELT 
Resource Package. 

o Appendix B: The Student Performanea r^^^i (SPIA nnrM^^n^ Also 
see Section 2, SPL Document, MELT Resource Package. 

o Appendix c: Section 3, Core Currieui im , Also see Appendix ill 
of the MELT Resource Package. 
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Chapter III 



MONITORING AND BVALUATIO'!; 

OVERVIBW 

Monitoring and evaluation provide for isystematic review and 
analysis of progran implementation and are ftindamental to the 
overall management plans of the funding agency and the contractor. 
The results of either activity may relate to various purposes, 
including: 

. Facilitating program improvement 
. Documenting the extent of goal achievement 
. Oetemining and documenting cost effectiveness 
. Future program planning and funding, and 
. Determining ELT program impact on the target population and 
on the broader goals of resettlement 

Although monitoring and evaluating are, from a conceptual 
standpoint, closely related functions, they are not identical. 
For the policymaker, the major emphasis will be on monitoring since 
more resources for this activity are available. Additionally, 
State and Federal regulations more often mandate monitoring while 
viewing evaluation as a discretionary activity. This chapter 
attempts to differentiate between these aspects of assessing 
programs and provides guidelines for both. 

RBC0MMBHDATI0H8 

1. h ■^ttitorlac and evaluation plan should b e a reonlrement 
ia responding to an bjp or in developi ng contraotual work 
BlABft* 

The purpose of monitoring is to assess program performance 
against projected goals and standardized performance 
indicators. The monitor asks two fundamental cpiestion^: Is 
the project meeting its stated objectives, and is it in 
conformity with grant/contract requirements? 

2. Monitoring m oonaiat ef wd««k w or "on-alteH reviews . 

Desk monitoring is an analysis based on review of program, 
fiscal, and statistical reports. Other aspects, including 
program management or program quality, require on-site 
monitoring and generally entail direct observation, review of 
case files, and examination of other internal records. 

Forms or instruments for collecting data should be developed. 
Tho««% for use in on-site monitoring may be shared with the 
projrams in advance of the visit. 

11 
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3- ypligVlXTa ahomd determine prloritiaa fo r information 
recreired for reporting _pttrpoaes ralativa to ny^^ ta^ q^fc 
P9P^lttiOB> — aoope mnd daaicm of lancmaae aarvieaa. 
atMdant ontcemea^ and * ,oat oonaiderationa . 

Policymakers must weigh the benefit to be gained from program 
reporting against the costs associated with generating that 
information. Extensive information gathering may cut into 
service monies. Conversely, to the extent that reporting 
requirements are reduced, program monitoring may need to be 
intensified, thereby draining program budgets. Thus 
policymsUcers should estzOslish reporting requirements after 
weighing the relative importance of: 

. Federal, State, and local requirements 

. information required to document progrsua activity, 

program cost, and program outcomes 
. The availability of time, staff, and/or funds 

Th» imaq—ent plan of aaeh ftmdl na of flea ahould apaetfv 
vendor reporting raaniramanta for the data to be 
COllect«d aad the frecmanev of rai portina. The contractor 
AIS — a resaonalbilitv for anaur ina compliance with 
contract coala. 

Contractors should be required to establish procedures to 
review performance on a continuing basis. Questions which 
may be useful in the monitoring process include: 

. Is the proposed target population the group which is 
actually being served? Needs to be served? 

. Are the types of instruction and curriculum materials 
specified in the proposal being implemented in the 
classroom? 

. Are there variances in instructional approaches for 

different tairget populations? 
. Is there a process which builds upon language assessment 

results in determining client's needs for services? 
. Is required documentation of program activity, cost, and 

outcomes being kept? 

Pytl 9h0M be collected for monitorinci kav aapaeta of 
ttf PWqrMIt program performance, ■aaa oament and fiacal 
^•tallt 

a. Program performance 

. Projected versus actual 

. Unduplicated clients served 

. Class size 

. Testing (quantitative changes) 

12 
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• Outcomes: completion and positive termination 



b. Management 

. Referral process/ interagency coordination 

• Client service planning (Employment Development 

Plan or EDP) 

• Internal data collection 

• External reporting: timeliness; data 

verification; (reliability and validity 
checks) 

• Eligibility determinations: alien status, 

length of residency criteria, public aid 
status, etc., as applicaOdle 

• File documentation: case notes, address, social 

security number r etc. 

• Staff qualifications: 

• Staff development/training: proposed vs actual 

• Supervision/staff performance review 

• Staffing configuration: proposed vs actual 

c. Fiscal Detail 

• Proposed versus actual cost per enrollee 

• Cost per contract hour 

• Cost per level 

• Cost per class 

• Testing costs 

• Outcome costs 

• Proposed v. actual cost per successful 

completion 

• Cost per positive termination (e«g«, dropped out 

to take a job) 

6* The State Coordinator's office should have overall 
reSBOnsibilitv for monitoring the proiact. 

When monitoring is performed by an outside source, the state 
Coordinator's Office should specify the monitoring 
objectives, type, and target sites. Documentation regarding 
the extent to which the stated goals and objectives were met 
should be incorporated into the final report. 

7« ggooram improvement should b e a focus of the monitoringr 

The monitor may require corrective action and/or provide for 
technical assistance where problems are identified. 

********* 
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«. Evaluation ahould foeua on: "la tha proiaet affectiva?" or 
«*Wha1: la tha lapaet bavond tha atatad obieotiva ?" 

Evaluations nay ba done by the state or an Independent 
party. The evaluation function should be used to obtain 
information which is beyond the scope of monitoring or which 
must be gathered independently of the funding source for 
reasons of potential bias. 

9« Tbe purpose and intendad audianca of th a evaluations ahould 
be Identified and the •valuation format and con^^mnt^ pl«""^«1 
aeeordinalv. 

The intended audiences for the evaluation report are usually 
policy and decision-makers at the local, state, regional, or 
Federal levels. The format in which the report is presented will, 
of necessity, differ for different audiences and purposes. 

Evaluation may be formative and summative. Formative evaluations 
focus on the quality and process of service delivery. Summative 
evaluations are outcome- focused, usually with heavy emphasis on 
costs. 

Consistent with the evaluation purpose, programs should collect 
and use data, such as student and instructor evaluations, follow- 
up surveys of former students, test results, and program 
outcomes. The primary use for evaluation results is in improving 
the quality of the services provided. The evaluation report can 
also be an effective tool in influencing decision-making at the 
State, regional and Federal levels. 

External evaluations should be periodic and cover one or more 
areas of concern. Questions which might be included in the 
program evaluation plan: 

. was the target population served? 

. were programs effective in meeting their goals and 

objectives? 
. were proposed client outcomes attained? 
. were multiple funding sources used in providing services 

to clients? 

. did programs collect the data needed to support their 

evaluation plans? 
. were the services cost effective? 

. did the ELT program contribute to the broader goals of the 
resettlement program? 



RESOURCES 

MELT Technical Assistance Packaqa. particularly the MELT-TA Self 
Evaluation Instruments for Programs (SEIP-ELT and SEIT-VELT) . 

14 
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Chapter IV 



C008DZ1IXTZN6 BLT AHD OTHER RESOURCES 



OVERVIEW 

To ensure its effectiveness, English J^nguage Training should be 
integrated into the service delivery system in such a way that it 
supports the goals of the self-sufficiency plan. To this end, 
there should be a clear assessment of all services available to 
refugees, both mainstream and ORR funded, and the interface of 
those services with refugee ELT programs. This chapter addresses 
methods for coordinating services and promoting communication-among 
individuals who represent the various aspects of the service 
delivery system. 



RECOMMEMDATZONS 

^- Thf aoOPe and aBDroDrlaimn«« a cf local resoureaa. both main- 
gtrtMl and refuaee-gpeolfie. shoM ld be aaaesaed to determine 
how best to use available ^iin<|«- 

Since some aspects of ELT are available through mainstream 
resources (e.g., adult basic education programs, community 
colleges, vocational training programs, public schools, and 
volunteer programs) , decisions are needed regarding the most 
appropriate mix of services, namely: 

a. Should ELT services be provided to targeted refugees 
using only refugee-specific funding? 

b. Should refugees be enrolled in mainstream ELT? 

c. Should ORR funding be used to support refugees in 
mainstream progrzuns? 

d. Should a combination of these options be selected? 
For example, ORR funding may be used for refugee- 
specific ELT for Student Performance Levels I - IV, * 
or for those on public assistance who have been in 
the country less than 18 months, while refugees 
outside these categories might be referred to 
mainstresuB programs. 



See Appendix B. 




As policymakers explore the various combinations, the goals 
of the mainstream programs should be examined to determine if 
they are consistent with the goals for the target population. 
Additionally, States may wish to examine how well a mainstream 
program cem incorporate refugees for purposes of: 

Data collection 
. Fiscal accountability 
. Differentiation of clients 
. Curriculum design 
. Participation by mandated clients 

Vh»rm sh ould hm m. clear and wll-defln#d relationship between 
— play— Bt- mBa^—nt. a nd gl.T aarvieea. 

If ELT services are to be an integral part of the service 
delivery system, Voluntary Agencies (V0LA6S) , social services 
agencies, job developers and ELT providers should communicate 
regularly regarding changes in the incoming refugee 
population, potential employment opportunities and 
requirements, and any significant change in the refugee 
resettlement climate. This communication should enable ELT 
providers to adjust curriculum in a timely manner, to modify 
class structure, lesson plans, and methods, and to be better 
prepared for a changing student clientele. Additionally, 
procurement documents for ELT services should request a 
specific plan from applicants regarding communication and 
information-sharing with other significant agencies in their 
area. 



etandardiaad atndant Pargor aanea Levels (SPL'a) and the Baale 
gnallsh SXills Teat (B.g.a.T. ), or an equally valid and 
rellabla aaaaaamant tool, ah ould be used across the aerviea 
dalivrv avatam to facilitata appropriate evaluation, aarviea 
Planning, referral, and -tab pl acaaant. 

Discussions amonr; case managers, job developers, employers, 
and teachers regarding a client's goals and his readiness for 
specific employment will be better informed if consistent 
measures, such as the B.E.S.T. scores and the SPL descriptions 
for language, are used to describe the client's language 
abilities. This coordination may also help to avoid 
duplication of effort and prevent competition between 
agencies. 

Technical assistance on standardized terminology, such as the 
SPL document, should be provided to case managers and 
'•mployment service providers as well as ELT programs so that 
they will understand the terminology and can apply it 
appropriately. 
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BLT providTs ahould hav incut into tha developm^Bt »nA 
PTiodie r«vi«w of el i ant STvica plana. 

Effective coordination between the various components of the 
service delivery system requires a process for sharing 
information regarding language needs and progress, job search, 
health care needs, and related issues. ESL teachers who work 
with clients on a daily basis should provide up-to-date 
student progress assessments to enploynent service providers, 
using standardized language (e.g. SPL's) familiar to case 
managers and employment program staff. 

WhW t Client «t Bnqligh language proficiency ha« tmmn 
idantlfied mm a sioi iif leant barrier to amplevment. regular 
PlrticiMtiffn in IPProPrif BLT programs mav ba ra guired for 
continued receipt of naigara hanafita. 

Within the context of fimding limitations and state/county 
welfare requirements, the state may wish to strengthen the 
relationship between paurticipation in language training 
programs and continued eligibility for cash and medical 
assistance. If elt is a state-mandated condition for 
receiving benefits, programs must be required to monitor 
attendance and to ensure that clients are receiving 
appropriate training. 

SB80USCE8 

Appendix A: Melt Resource Package 

Basic English skills Test (B.E.S.T.) Manual. Correlations between 
the B.E.S.T. scores and SPL's are included for both listeninc and 
speaking skills and for literacy skills in the Testing Appendix 
Section IV. 

Guidelines for curriculum development, sample needs assessments and 
lesson plans are included in Section 3 of the MELT Resources 
Package. 

MELT Technical Aaslatanea Rea ource Paelca^ye 

Workshop materials and activities for training case managers, job 
developers, and employers in the use of the SPL's, the Core 
Curriculum Document and the B.E.S.T. are included in the MELT-TA 
Resource Package. 

MELT Technical Asa ^ stance Paclcacpe 

The MELT TA Package contains resource information and a workshop 
format for the design, contracting, monitoring, and evaluation of 
refugee language training programs. Also see The MELT- TA Self- 
Evaluation Instruments for Programs (SEIP-ELT and SEIP VELT) . 
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CHAPTER V 



C0MSZ0ERIM6 COSTS 

OVBRVISW 

The responsibility of the policymaker is to allocate resources 
efficiently when purchasing necessary services. The issue of cost 
permeates the entire process of planning, procurement, monitoring 
and evaluation. For the purpose of planning, the policymaker 
evaluates applicants' projected costs and selects the most cost 
effective and efficient applicants for funding. in order to 
monitor or evaluate programs effectively, the policymaker compares 
actual expenditures with the projections individually or system- 
wide* 

Because each system has unique clients and programmatic design, 
this section assumes that the best approach to cost analysis is a 
process based on the system's historical data and is appropriate 
to the structure of the programs, while LLare are many ways of 
approaching and defining costs, a policymaker should evaluate 
models and adapt the one best fitted to the needs and circumstances 
of the State or locale. 



RSCOIOIBHDATZONS 

1. gWtra9Uqa M<l BJII l <<q1na BLT programs. Bolievmalrar« should 

irtiOttlaf olear definitiona of u llo^able eamta and aaaaaa 
Pgynria ICqOMPlishaeBta on Hha haa ia of bQth proeeaa and 

In considering costs, two elements are critical: Types of cost and 
imua of cost. For purposes of determining allowable costs, the 
policymaker should comply with State procurement procedures using 
established definitions of direct and indirect costs. These 
definitions should be made available to vendors in the procurement 
and contract development phases of the process. 

The second element, imita of cost, are measures of process or 
specific outcomes. 

A process cost is generally a description of the service itself. 

Examples of process cost measures are: 

. cost per participant 

. cost per class 

. cost per contact hour of instruction 

. cost per client per hour of instruction 
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Outcome costs relate to the products of the program. 

Exeuaples of outcome costs are: 

. cost per positive termination 

. cost per successful completion of a course 

. cost per successful achievement of a designated benchmark 

Policymakers must choose unit cost measures which are most 
appropriate for program planning, procuring, and monitoring. 
Criteria for selecting unit measures of cost are: 

. The unit should be applicable across programs. 

. There must be specific definitions for measures selected 

. The unit must provide program accounted) ility. 

. The unit must be readily reported. 

2* A POliCTmaker should develop a range of allovable process and 
outcome costs to be uaed in p^li^^n^T^ ^, proeurina. and 
JBCnitorinc since absolutes have ll■l^ ted aoplieabilltv. 

Examples of calculating formulas, requirements for using them and 
considerations in selecting a formula can be found in Appendix F. 
Process and outcome costs should )e analyzed in order to provide 
an overall picture of actual costs. 

Each system has a unique client population, goals, and service 
configurations. Therefore, each system should identify ranges of 
cost which are based on actual data from within the system. 

The policymaker should use cost information to target levels of 
funding during the planning phase, to negotiate realistic contracts 
during the procurement phase, and to compare actual expenditures 
with projections during the monitoring p^iase. 

. Considerations in developing a cost rzmge should include: 
Factors Not Controllabl e bv Vendor 

• . Community standards of wages and benefits 
. Community standards of rent, utilities, and maintenance 
. Level of ESL acquisition barriers within the targeted 
population 

Factors Controllable bv Von^^^y 

. Size of client base to be served 

. Staff qualifications and configuration 
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3* Policvmakera ahould •neoin;'^ oa th« uaa of mainstraan fxinda to 
wcmant rafuaaa apaoific dollar a in tha nroviaion of BI.T. 

Many agencies interested in and capable of providing English 
language training to refugees also receive funding from other 
sources, (e.g., school districts, conmunity colleges, and private 
educational institutions) . When other funding sources are present, 
a pollcynaker must ensure that the vendor's cost allocation system 
appropriately allocates both direct and indirect coses to the 
refugee funded program to ensure that refugee funding is not being 
used to cover costs of non-refugee programs. 

The following elements should be identical for accounting and 
accountability purposes, vendors should: 

. Use the same factors in distinguishing direct and indirect 

costs across funding sources. 
. Be required to pro-rate. Exan^le: 

$40,000 « ORR 

$60,000 * Other sources 

40/60 split on direct and indirect costs 
. Use the prorated formula described above as the basis of 

reimbursement across funding sources 
. Use the same cost per unit of completion across funding 

sources 

. Use the same outcome and process measures across ftinding 
sources, (e.g., definition of completion) 

4* At»M» the available modala of iP>4»hi«'««B ^nt. ehooaa ti»«t 
which baat w^iita a proagaaia niatrative reaouroea and tha 
vendors used aa wall aa i^hat which proiote a analltv. 

Typically, policymakers choose one of two broad models of 
contracting for payment: actual cost reimbursement or performance- 
based reimbiirsement. 

a. Actual goat Paiwhnraement; Based on a negotiated line item 
budget, vendors submit monthly or quarterly statements of 
expenditures, usually in line item form. After review of 
appropriate support documentation, and if expenditures do not 
exceed the agreed upon budget, payment is made. Some funders 
allow an average of 10-15% on any given line item as a margin 
for budget management, as long as the aggregate expenditures 
for the reporting period are within budget and the cumulative 
year-to-date escpenditures are following projections. 

As with any method of contracting, the cost reimbursement 
model has advantages and disadvantages: 

Advantages: 

. Makes regular cash flow available to the vendors. This is 
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particularly important for small organizations, such as MAAs 
VOIAGs or small public institutions. 



. May provide advances for program start-up to be recouped in 
the latter part of the fiscal vear. This is an advantage for 
small organizations. 

. Provides full accountability of how program dollars are 

spent, including some knowledge of program design (e.g.. 
staffing pattern) . 

. Can provide an incentive to agencies to serve a ooDulation 
of "difficult clients". 

. Does not promote "creaming' from among clients if the 
program targets a heterogeneous population. 

Disadvantages : 

. Payments are tied to process rather than outcome. 

. There is difficulty in adjusting the funding level 
commensurate with reductions or changes in the client 
base. 

. If a program proves unworkable or unproductive, it is more 
difficult to implement timely contract termination and thus 
salvage resources. 

. A greater commitment of resources to fiscal administration is 
required - both on the part of the fimder and the vendor. 

^* Perfomance-Based Contrafstipg; Based on a comprehensive 
program plan and a detailed analysis of all costs involved in 
operating the program, concrete outcomes are negotiated and 
a unit price established. A maximum reimbursement level (i.e. , 
total face value of the contract) is established as well. 

Benchmarks for program phases and outcomes are determined with 
clear delineations of reimbursement upon achievement. A 
representative sample of agreements reached might include: 

. Contract period: July 1, I98b - June 30, 1989 
. 200 clients to be enrolled in Course A 
. 160 clients to complete Course A (80%) successfully 
. Negotiated Unit Reimbursement per enrollment in course A 
" $80.00 

. Negotiated Unit Reimbursement per completion of Course A 
" $400.00 

. Maximum Level of Reimbursement for Course A = $96,000 
. Payment Schedule ■ up to 20% of contract value upon 

enrollment of 200 clients ($19,200) or appropriate pro- 
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rata at agreed upon date of termination of enrollment 
(8/31/88) ; up to 80% of contract value upon i'uccessful 
client completions ($76,800) or the appropriate pro-rata 
at agreed upon date of program termination (6/30-89). 

.Op to $19,200 or 20% of the face value of the contract may 
be provided as forward funding as of 7/1/88. This 
anoiuit will be recouped as follows: 20% ($3,800) against 
enrollment reinbursenent and 80% ($15,360) against client 
completion reimbursement at any time prior to 60 days of 
contract closure (4/30/89) . 

Again there are advantages and disadvantages to the Performance- 
Based model: 



Advantages: 



Dollars are tied to concrete outcom es. 

Renegotiation of contracts is reasonably timely if changes 

occur in the client base. 
Advances may be provided for program start-up and recouped 

in the latter part of the fiscal year— is a must for 

small organizations. 



Disadvantages: 



. Irregular cash flow and the risk of losing money due to 
failure to perform may dissuade smaller or "newer" 
organizations from service prov^-^ion. 

. There is little accountability of uow program dollars are 
spent. 

. 'TIS model discourages services to the comprehensive client 
base and to "difficult" clients since the model promotes 
"creaming" . *^ 

. Little insight into the quality of service is given. 

. The provision of appropriate support services is 
discouraged. 

. No opportunity to assess program shortcomings and 

provide technical assistance and/or corrective action 
is provided. (For instance, high staff turn-over is 
apparent.) 
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EDITOR* S EPTLOGUE 



This dociunent represents the collective ^ixperience 
and thinking of tixe MELT Work Group regarding 
effective ways of teacliing English to refugees and 
of Staking the ELT program a partner in a 
conprehensive service delivery network. Although 
nany of the iteas in this document speak directly 
to the program configurations amd priorities of 
the refugee program, the members of the work Group 
feel that most of the ELT program content 
guidelines will be appropriate for any quality ELT 
program for adults, whether for refugees, 
immigrants, aliens newly legalized under the 
amnesty program, or other adult learners of 
English. 

Clearly, the most important goal of an ELT program 
is to enable the students to learn the English they 
need to aid in building successful lives, students 
have been learning under many different methods 
with a wide variety of instructors, and this will 
undoubtedly continue to be the case. These 
guidelines axB not presented as definitive but are 
intended to aid policymakers, program 
administrator, and service providers in their 
continuing efforts to be effective in helping 
refugees to help themselves. 



spring Institute for International studies 
Myma Ann Adkins 
Barbara J. Sample 
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The MELT Resource Package 



The MEiT Re?5ource Package is designed to be utilized as a whole 
or In part by a wide range of programs and service providers. 

1. Student Performance Levels 

The SPL document Is a set of descriptions stating what students 
should be able to accomplish with their language skills at ten 
different levels. Each level la described In terms oi a 
student's listening, speaking, reading, and writing skills; 
ability to communicate with a native speaker; and readiness for 
6.-ployment. The descriptions do not rely on technical language 
and are hence comprehensible by non-ESL professionals. The SPL 
Section of the MELT Resource Package includes, m addition to 
the document Itself, a pronunciation rating scale, a range of 
contact hours needed for level changes, and student and program- 
related conditions which affect level gain. 

The SPL Document Is an Integral part of the MELT Resource 
Package. The SPL's are correlated to Core Curriculum Instruc- 
tional Levels and the B.E.S.T. score ranges. Use of the SPL 
Document Is not, ho««ever, dependent on the total MELT Resource 
Package. Anyone needing to describe or better understand a 
student's language proficiency may find the materials useful. 

See Appendix 2 for the complete SPL Document. 

2. Core Curriculum Guide 

The Core Curriculum Guide i« "competency-based". A competency 
Is defined as a demonstrated ability to perform a llfe-skllls 
task using language. The Curriculum document lists competencies 
in eight topical and seven cross-topical areas for seven levels 
Of instruction. These Instructional levels correlate to the 
first seven of the Student Performance Levels and represent the 
life skills competencies needed to attain self-sufficiency. 

In addition to the Core Curriculum Qulde itself, the Curriculum 
Section of the MELT Resource Package Includes examples and 
guidelines for developing and adapting curriculum to meet luctl 
program needs and goals. 

The Core Curriculum Section, like the SPL Section, Is an 
Integral part of the MELT Resource Package, yet It may be used 
independently, it is Intended to provide guidelines and 
assistance In competency-based ELT curriculum development and 
should be adapted to local program needs and goals. 
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3. Basic English skills Test (3.E.S.T.) 



The B.E.S.T. Is a test of elementary listening coirirehenslon, 
speaking, reading, and wltlng. It Is Intended for use with 
llmlted-Engllsh-speaklng adults for whom Information on the 
attainment of basic functional language skills Is needed. The 
test consists of two sections: a Core section and & Literacy 
Skills section. 

The Core section of the B.E.S.T. Is an individually administered 
Interview which requires 15 to 20 minutes per person. It 
Includes real-life tasks which require listening and speaking, 
such as handling money, following directions, telling time, and 
conversing socially. Also, reading sight words and completing a 
short biographical data form together serve as a screening 
device to identify students for whom the Literacy section would 
be appropriate. Pronunciation Is evaluated throughout the" 
administration of the Core section, and a global rating is 
given. 

The Literacy section, which may be administered either 
individually or on a group basis, presents a variety of reading 
tasks ranging from recognizing dates on the calendar and under- 
standing food and clothing labels to reading bulletin announce- 
ments and newspaper want-ads. Writing tasks range from 
addressing an envelope and writing a rent check to filling out 
an application form and writing a short passage on a biographi- 
cal topic. Testing time for the Literacy section is one hour. 

The B.E.S.T. is designed to provide useful information in three 
basic areas: 

(1) evaluating the extent and nature of student's En^rllsh 
language proficiency on entry into language training 
courses, for purposes of appropriate class placement or for 
planning individualized learning activities best suited to 
a given student. 

(2) determining the progress o£ individual students, or 
the class as a whole, in developing functional proficiency 
in English with respect to the types of "survival'" and pre- 
vocatlonal language-use situations represented in the tost. 

(3) providing diagnostic feedback concerning students' 
acquisition or lack of acquisition of each of the 
particular language-use tasks Included in the test (for 
example, telling time, dealing with money, etc.). This 
information may be used for overall course planning or 
Individual remedial instruction. 
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APPENDIX B 



STtTOBNT PERFORMANCE LEVEL DOCUMENT 
INCLUDING THE ABBREVIATED VERSION 
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STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 



Q No ability whatsoever. 



LISTENING 
COMPREHENSION 



No ability whatsoever. 



ORAL 

COMMUNICATION 



No ability whatsoever. 



READING 



No ability whatsoever. 



WRITING 



B.E.S.T. 
SCORE 



No ability whatsoever. 



0*8 



I« Functions minimally, 
if at all. in English. 

• Can handle only 
very routine entry* 
level jobs that do not 
require oral commu* 
nication. and in which 
ail tasl(s can be easily 
demonstrated. 

• A native English 
spealcer used to deal* 
ing with limited English 
spealcers can rarely 
communicate with a 
person at this level 
except through 
gestures. 
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Understands only a 
few isolated words, 
and extremely simple 
learned phrases. 
(What's your name?) 



• Vocabulary limited to 
a few isolated words. 

• No control of 
grammar. 



• Recognizes most 
letters of the alphabet, 
and single*dlglt 
numbers. 



Copies letters of the 
alphabet, numbers, 
own name and ad- 
dress; needs assis- 
tance. 



9-15 
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STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 


LISTENING 
COMPREHENSION 


ORAL 

COMMUNICATION 


READING 


WRITING 


a.E.S,T. 
SCORE 


TT • Functions in a 
11 very limited way in 
situations related to 
imnriediate needs. 

• Can handle only 
routine entry-level 
jobs that do not re* 
quire oral conrinriunica* 
tion, and in which all 
tasks can be easily 
demonstrated. 


• Understands a limited 
number of very 
simple learned 
phrases, spoken slow- 
ly with frequent 
repetitions. 


• Expresses a limited 
number of Immediate 
survival needs using 
very simple learned 

phrases. 

• Asks and responds to 
very simple learned 
questions. 

• Some control of very 
basic grammar. 


• Recognizes letters of 
the alphabet, numbers 
1-100, and a few very 
common sight words 
(e.g. name, address, 
stop). 


• Writes letters of the 
alphabet, numbers 
1-100, very basic 
personal Info, on sim- 
plified forms; needs 
assistance. 


16-28 . 


• A native English 
speaker used to deal* 
ing with limited English 
speakers will have 
great difficulty conv 
municaiing wiin a 
person at this level. 
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STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 



III 



• Functions with 8ome 
difficulty in situations 
reiated to immed!att 
neads. 

• Can handie routina 
entry-lavai jobs that 
invoive only the most 
basic orai communi* 
cation, and in which 
all tasl<s can be 
demonstrated. 

• A native English 
speaker used to deal- 
ing with limited Eng^ 
iish spealiers will have 
great difficulty 
communicating with a 
person at this level. 
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LISTENING 
COMPREHENSION 

Understands simple 
learned phrases, 
spolten slowly with 
frequent repetitions. 



ORAL 

COMMUNICATION 

• Expresses Immediate 
survival needs using 
simple learneci 

phrases. 

• Asks and responds to 
simple learned 
questions. 

• Some control of very 
basic grammar. 



READING 

• Reads and understands 
a limited number of 
common alght words, 
and short, simple 
learned phrases re- 
lated to immediate 
needs. 



WRITING 

• Writes a limited num- 
ber of very common 
words, and basic per- 
sonal info, on sim- 
plified forms; needs 
assistance. 



B.E.S.T. 
SCORE 

29-41 
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GENERAL LANGUAGE 
ABILITY 



LISTENING 
COMPREHENSION 



STUDENT PERFORMANCE LEVELS 

READING 



ORAL 

COMMUNICATION 



Writing 



B.E.S.T. 
SCORE 



IV 



• Can satisfy basic 
survival needs and a 
few very routine 
social demands. 

• Can handle entry- 
level jobs that involve 
some simple oral 
communication, but 
in which tasks can 
also be demonstrated. 

• A native English 
speaker used to deal* 

ing with limited English 
speakers will have 
difficulty communi* 
eating with a person 
at this level. 



39 




• Understands simple 
learned phrases easily, 
and some simple new 

phrases containing 
familiar vocabulary, 
spoken slowly with 
frequent repetitions. 



• Expresses basic sur* 
vival needs, including 
asking and responding 
to related questions, 
using both learned 
and a limited number 
of new phrases. 

• Participates in basic 
conversatkms in a 
few very routine 
social situations (e.g. 
greeting, inviting). 

• Speaks with hesitation 
and frequent pauses. 

• Some control of 
basic grammar. 



• Reads and understands 
simple learned sen* 
fences and some new 
sentences related to 
immediate needs; 
frequent misinter« 
pretations. 



Writes common words 
and simple phrases 
related to immediate 
needs; makes frequent 
errors and nc;eds 
assistance. 



42-50 
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STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 



LISTENING 
COMPREHENSION 



ORAL 

COMMUNICATION 



READING 



WRITING 



B.E.S.T. 
SCORE 



V 



• Can satisfy basic sur- 
vival needs and some 
limited social 
demands, 

• Can handle jobs 
and job training 
that involve following 
simple oral and very 
basic written Instruc* 
tions but in which 
most tasks can also 
be demonstrated. 

• A native English 
speaker used to deal* 
ing with limited English 
speakers will have 
some difficulty com* 
municating with a 
person at this level. 



• Understands learned 
phrases easily and 
short new phrases 
containing familiar 
vocabulary spoken 
slowly with repetition. 

• Has limited abilitv to 
understand on \t\^ 
telephone. 



• Functions indepen* 
dently in most face* 
to*face basic survival 
situations but needs 
some help. 

• Asks and responds to 
direct questions on 
familiar and some 
unfamiliar subjects. 

• Still relies on learned 
phrases but also uses 
new phrases (i.e. 
speaks with some 
creativity) but with 
hesitation and pauses. 

• Communicates on the 
phone to express a 
limited number of 
survival needs* but 
with some difficulty. 

• Participates in basic 
conversations in a 
limited number of 
social situations. 

• Can occasionally 
clarify general mean* 
ing by simple re* 
wording. 

• Increasing, but Incon* 
distant, control of 
jasic grammar. 



• Reads and understands 
some short simplified 
materials related to 
basic needs with some 
misinterpretations. 



• Writes phrases and 
some short, simple 
sentences; completes 
simplified forms. 

• Ma 's some errors: 
neeus assistance. 



51-57 
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17T • wan satisfy most 
V 1 survival nteds and 

limllad social 

demands. 

• Can <andle jobs and 
Job Iral/iing thai 
involve following 
simple oral and writ* 
ten instructions and 
diagrams. 

• A native English 
speaker not used to 
dealing with limited 
English speakers will 
be able to communi- 
cate with a person at 
this level on familiar 
topics, but with dif- 
ficulty and some 
effort. 



• Understands conver- 
sations containing 
some unfamiliar 
vocabulary on many 
everyday subjects, 
with a need for re- 
petition, rewording 
or slower speech. 

• Has some ability to 
understand without 
face-to-face contact 
(e.g. on the telephone. 
TV) 
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STUDENT PERFORMANCE LEVELS 



ORAL 

COMMUNICATION 



READING 



WRITING 



B.E.S.T. 
SCORE 



• Functions Indepen- 
dently in mo»t sur- 
vival situations, but 
needs some help. 

• Relies less on learned 
phrases; speaks with 
creativity, but with 
hestltatior?, 

• Communicwies on the 
phone on familiar 
subjects, but with 
some difficulty. 

• Participates with some 
confidence in social 
situations when 
addressee^ directly. 

• Can sometimes clarify 
general meaning by 
rewording. 

• Control c. oasic 
grammar evident, but 
inconsistent; may 
attempt to use more 
difficult grammar but 
with almost no control. 



^ Reads and understands 
simplified materials on 
familiar subjects. 

• May attempt to read 
some non-simplified 
materials (e.g. a notice 
from gas company), but 
needs a great deal of 
assistance. 



• Performs basic writing 
tasks in a familiar 

context including 5hort 
personal notes and 
letters (e.g. to a 
teacher or landlord). 

• Makes some errors; 
may need assistance 



58-64 
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ON 



STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 


LISTENING 
COMPREHENSION 


ORAL 

COMMUNICATION 


HEADING 


WRITING 


B.E.S.T. 
SCORE 


\7TT* Can satisfy survival 
y 11 needs and routine 

work and social 

demands. 

• Can handle work that 
involves folk>wing oral 
and simple written 
instructions in familiar 
and some unfamiliar 
situations. 

• A nauve English 
speaker not used to 
dealing with limited 
English speakers can 
generally communi* 
cate witli a person at 
this level on familiar 
topics. 


• Understands conver* 
sations on most every- 
day subjects at normal 
speed wher« addressed 
directly; may need 
repetition, rewording, 
or slower speech. 

• Understands routine 
work*related conver- 
sations. 

• Increasing at>ility t;o 
understand without 
face«ic*face ccntact 
(telephone. TV. radio). 

• Has difficulty following 
conversation between 
native speakers. 


• Functions indepen- 
dently in survival and 
many social and work 
situations, but may 
need help occasion- 
ally. 

• Communicates on the 
phone on familiar 

subjects. 

• Expands on oaslc 
ideas in conversation, 
but still speaks with 
hestltatlon while 
searching (or appropriate 
vocabulary and grammar. 

• Clarifies general 
meaning easily, and 
can sometimes convey 
exact meaning. 

• Controls haftic Oram* 
mar. but not more 
difficult grammar 


• Reads and partially 
understands some 
non-simplified 
materials on everyday 
subjects; needs 
assistance. 


• Performs routine 
writing tasks within a 
familiar context. 

• Makes some errors; 
may need assistance. 


65-t- 
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STUDENT PERFORMANCE LEVELS 



GENERAL LANGUAGE 
ABILITY 


LISTENING 
COMPREHENSION 


ORAL 

COMMUNICATION 


r ' 

READING 


WRITING 


B.E.S.T. 
SCORE 


1 rTTT • Can participate effec* 
Ylli lively in social and . 

familiar worl( situa- 

tions. 

• A native English 
speaker not used to 
dealing with limited 
English speakers can 
communicate with a 
person at this level on 
almost all topics. 


• Understands general 
conversation and con* 
versation on technical 
subjects in own field. 

• Understands without 
face^to-face contact 
(telephone. TV. radio); 
may have difficulty 
following rapid or 
colloquial speech. 

• Understands most 
conversation between 
native speakers; may 

miss details if speech 
is very rapid or collo- 
quial or if subject is 
unfamiliar. 


• Participates effectively 
in practical and social 
conversation and in 
technical discussions 
in own field. 

• Speaks fluently in both 
familiar and unfamiliar 
situations; can handle 
problem situations. 

• Conveys and explains 
exact meaning of 
complex ideas. 

• Good control of 
grammar. 


• Reads and understands 
most non-simplified 
materials including 
materials in own field. 


• Performs wniing tasks 
with reasonable ac- 
curacy to meet social 
and basic work needs 




T Y * Can participate 

1 A fluently and accurately 
in practical, social, 
and work situations. 

• A native English 
speaker not used to 
dealing with limited 
English speakers can 
communicate easily 
with a person at this 
level. 


• Understands almost all 
speech in any context. 
Occasionally confused 
by highly colloquial or 
regional speech* 


• Approximates a native 
speaker's fluency and 
ability to convey own 
ideas precisely, even 
in unfamiliar situations. 

• Speaks without effort. 

• Excellent control of 
grammar with no ap- 
parent patterns of 
weakness. 


• Reads non«simplified 
materials. 


• Approximates a 
native speaker's 

ability to write 
accurately. 




Y * Ability equal to that 
A of a native speaker 
of the same socle* 
economic level. 


• Equal to that of a 
native speaker of the 

same socio-economic 
level. 


• Equal to that of a 
native speaker of the 

same soc*o-economic 
level. 


• Equpi to thai of a 
native speaker of f*'^ 
same socio-economic 
level. 


» Equal to that of a 
native speaker of the 
same socio-economic 
level. 


1 

00, 
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PRONUNCIATION: GLOBAL RATING B-9 



Speech is almost always unintelligible. 


0 


Speech is frequently not comprehensible. 


1 


Speech is generally understandable, but 
occasionally difficult or impossible to 
comprehend as a result of oronunciation 
problems. 


2 


Speech is readily understandable 
(from a pronunciation standpoint). 


3 
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STUDENT PERFORMANCE LEVELS - ABBREVIATED VERSION 



Q No ability whatsoever. 






T • Functions minimally, 
if at all. in English. 

• 


• Can handle only 
very routine entry- 
level jobs that do noi 
require oral commu- 

iiii#ciiiuii. aiiQ In wnicn 

all tasks can be easily 
demonstrated. 


• A native English 
speaker used to deal- 
ing with limiied English 
speakers can rarely 
communicaie with a 
person at this level 
except through 
gestures. 


TT • Functions in a 

v«ry Itmittd way in 
situations i9lated to 
immtdiatt nttds. 


• Can handle only 
routine entry«level 

jobs that do not re- 
quire oral communica- 
lion, and in which all 
tasks can be easily 
demonstrated. 


• A native English 
speaker used tu deal- 
ing with limiied English 
aoAakers will hawA 
great difficulty com- 
municating with a 
person at this level. 


TIT * Functions with soma 
difficulty in situations 
ralatad tolmmadlata 
naada. 


• Can handle routine 
entry*level jobs that 
involve only the most 
basic oral communi- 
cation, and in which 
ail tasks can be 
demonstrated. 


• A native English 
speaker used to deal- 
ing with limited Eng- 
lish soukafs will havA 

great difficulty 
communicating with a 
person at this level. 


TV * Can satisfy basic 
^ * survival naads and a 
few vary routine 
soc*il demands. 


• Can handle entry* 
level jobs that involve 
some simple oral 

communication, but 
in which tasks can 
also be demonstrated. 


• A native English 
speaker used to deal- 
ing with limited English 
speakers will have 
difficulty communi* 
eating with a person 
at this level. 


\F • Can satisfy basic sur* 
V vival needs and some 

limited social 

demands. 


• Can handle jobs 
and Job training 
that involve following 
simple oral and very 
basic written instruc- 
tions but in which 
most tasks can also 
be demonstrated. 


• A native English 
speaker uaed to deal- 
ing with iimited English 
speakers will have 
some difficulty com- 
municating with a 
person at this level. 
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17T • Can satisfy most 
V 1 survival naads and 

llmlfftd ftoclal 

demands. 


• Can handle jobs and 
job training that 
involve following 
simple oral and writ- 
ten instructions and 
diagrams. 


• A native English 
speaker not used to 
dealing with limited 
English speakers wilt 
be able to communi- 
cate with a person at 
this level on familiar 
topics, but with dif* 
ficulty and some 
effort. 


\ 7TT • Can fiAli*^^ fturvlt/fli 
mf I I wail 90113 f •uivival 

^ y^y^ needs and routine 
woric and social 
demands. 


• uan Handle work that 
involves following oral 
and simple written 
instructtons in familiar 
and some unfamiliar 
situations. 


• A native English 
speaker not used to 
dealing with limited 
English speakers can 
generally communi* 
cats with a person 
this level on familiar 
topics. 


VTTT ^ ^'^^ pariicipate effec- 
Y 111 lively in social and 

familiar worIc situa* 

tions. 




• A native English 
speak.^' not used to 
dealing with limited 
English speakers can 
communicate with a 
person at this level on 
almost all topics. 


TV • Can participate 

fluently and accurately 
in practical, social, 
and work situaiions. 




• A native English 
speaker not used to 
dealing with limited 
English speakers can 
communicate easllv 
with a person ai this 
level. 


Y • Ability equal to that 
^ of a native speaker 
of the same socio* 
economic level. 
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APPENDIX C 
CURRICULUM DEVELOPMENT AND ADAPTATION 
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C-1 

CURRICULUM DEVELOPMENT IN A COMPETENCY^BaSED ELT PROGRAM 

Inslruclional conleni in a compelency-based ELT program focuses on the application of basic 
language skills to life skills compeloncies as determined by an assessment of the learner's needs, 
goals, and abilities. In addition, characteristics unique to local education programs and com- 
munities affect the development and implementation of a curriculum. Before an ELT program 
develops a curriculum, it should identify those characteristics unique to the local students, com- 
munity, and program. 

Listed below are general questions to assist a program in specifying its individual characteristics. 
Students 

« What is the population to be served? (Numbers, educational background, 
ethnicity, etc.) 

What are the goals of the student population? (Employment, vocational 
training, GEO, etc.) 

• What are the current abilities of the student poptilation? (Language skills, 
technical skills) 

Communily 

• What are the cSaracteristics of the community? (Ci!y. small town, rural; 
multi-ethnic/multi-lingual, etc.) 

» What is the local job market? (Unemployment rate, types of jobs available. 
«»3lary levels, etc.) 

Proijram 

• What are the program's goals? (Rnding immediate employment for students, 
preparing sludents for entry into vocational skills training, preparing students 
for generat self-sufficiency in the community, etc.) 

• What services are the program funded to provide? (GeneraS ESL. VESL, cul- 
tural orientation, job development, etc.) 

• What is the program organization? 

Intensity and scheduling of the classes? (3 hours. 1 lime per week for 12 
weeks, etc.) 

Entry/exit criteria? (open/closed intake, completion schedule) 
Number of instructional levels? (2,3, etc.) 
Class size? (20, 15, etc.) 

Facilities and equipment? (classroom space, books, tape recorders, etc.) 

• What is the. staffing pattern? (Curriculum specialists, full-time teaching positions, 
administrative positions, etc.) 

• Who are the teaching staff? 0>atned and experienced ESL teachers, bilingual 
I Jss, volunteers, etc.) 

» On what outcome ^ Is the program evaluated? (Number of job placements, 
number of clients sucrsssfully completing a level, etc.) 

Answering these questions will identify the program characteristics and assist in determining the 
focus, scope, and content Oi he local curriculum. 
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CHART A: 
Curriculum Development/Adaptation Process 



c-2 



IDENTIFY language needs of students In the 
U.S. work environment and local community 

COMPARE the needs assessment results with the 
Core Curriculum and local curriculum topics 

SELECT or ADAPT topics 



COMPARE the competencies In the Core Curriculum document and 
the local curriculum with the needs {.isessme it resuKs 



SELECT OR ADAPT competencies 
and assign them to levels 



DEVELOP a cour-^e design by determining for each competency: 
• the commnnlcatlon function(s) 
• the graii...iatical structures 



DESIGN lessons 



DESIGN a plan for evaluating 
student achievement 
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SELECTING AND/OR DEVELOPING ASSESSMENT INSTRUMENTS 



57 



STUDENT ASSESSMENT IN A COMPETENCY-BASED ELT PROGRAM 

One of the advantages of using a compeioncy-based approacli :o language leaching is 'hal 
assessment Is built-in. The task of assessing what the students have learned Is based on previously 
specified reaMife competency objectives. Testing instruments for placemen! and achievement 
directly relate to the curriculum in the focaf instructional program. 

Selecting and/or Developing Assessment Instruments 

Competency-based ELT progra ms (or refugees focus nn the teaching of life skiffs and on the basic 
language sldns necessary to perform the Nfe sKMIs. Coi^efciafly available ESL language tests 
may b? reliable and valid for testing basic language sidlls but may not relate dfrectly to the locaf 
program's curriculum. Life sKHts tests^ appropriate lor aduft refugees are not readily avaHabte or 
may not be directly related to an ELT programme curricuhmi. Thus, in developing %f assessment 
system foi an ELT program, the local program win need to select and/or develop te«ts that are ap- 
propriate to competency-based ELT curricula and more specifically to the local programme ELT 
curriculum. 

In order to accomplish tiiis task, decisions wilt have to be made as to: 

1. The criteria iur movement or advancement within and exit from the local pro- 
grams: 

• attainn.^nt of corrpetencies 

• attainment of key competencies for a specific level 

• a determined s?ore on a test or tests 

• communicative ability 

• instructor's subfectivo judgment 

• a combination of two or more of the above factors 

2. The tesi(s) content: 

• attainment of competencies 

« aural/oral skills (communicalive ability, pronurKialion) 

• literacy/reading/writing skills 

• grammatical structures 

• a combination of two or more of the above factors 

3. Time and staffing constraints affecting the assessment process: 

• length of the test 

• method of testing: paper aruJ pencil, oral interview, applied performance/ 
task demonstratkm, instructor observation, combinatton 

of mettK)ds 

• test adminislratton 

4. Required recordkeeping: 

• test scores 

• competrncy check Hsts 

• student/class profiles 

• a combinatk>n of two or more of the above 

5. Feedback provided (f>ow and to whom): 

• students 

• administrators 

• funding agency 

• community 

These k>cal ELT program decisions assist in determining what testing instrument should be used 
and when. Usually a combination of commercially and k)cally produced tests is found to be the 
most appropriate. 
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E-1 



Partial List of Language Proficiency Testa 



Following Is a partial list of tests which are currently 
available In addition to the B.E.S.T. and which could be used to 
determine language proficiency level. The list Includes the 
naae of the test; the publisher, and a brief description. Two 
other resources which list ESL language tftsti* are the MELT 
Resource Package and Review of English Language Proficiency 
Testa, a TESOL publication by Alderson, Krahnke and Stansfield. 
Th«» latter resource Includes an extensix r description and 
evaluation of each test dealth with in the book. 
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Test 



Publisher 



Description 



Conpcehensive Adult Student 
Assessirant System (O^SAS) 



California Department 
of Education 



Can be used as a placement, diagnostic, actiievement, 
or certiCication test. Skills tested Include life 
skills and prevocational skills in the conte:<t oc 
reading and listening. There are multiple forms with 
score ranges correlated to SPL I-VII. 



English language Skills 
Assessment (ELSA) 



English as a Second Language 
Oral Assessment (ESlOA) 



HELP Test 



liyin Oral Interview/ 



John/Fred Test 



Secondary Level Ehgiish 
PLCficiency Tast CSLEE") 



Newbur/ House 



Literacy Vblunteer 
of Ainerica 



Designed to measure reading ^ility for placerrent. 
The test can also be used bo measure pre-post-test 
gain. There are three levels of difficulty with 
tMO fonns at each level. Ttet is easy to administer 
and takes 2S to 30 minutes. 

Can be used as a placement/ diagnostic^ and progress 
test. It is divided into four levels which indicate 
oral/aural proficiency In following directions and 
using specific Ekiqlish patterns and basic vocabulary. 



Alemany Press 



A screening instrvment for students with very basic 
Ekiglish skills to identify literacy and oral levels 
and to facilitate placement. Individually administered 
with no time limit. Scoring is based on cannunica- 
tion rather than grannatical corrections . 



Newbury House, 
Harper and Bow 



Designed to assess fitgrlish oral proficiency Cor 
diagnostic purposes. Scores correspond to five le'/els. 
There are three forms « * Individual, adtaiinistration 
takes S to 30 minutes. 



Language Innovations, 
Inc. 



An oral placement test which bests answering questions, 
asking questions and producing connected discourse. 
There are two forms (John and Fred) plus a short 
version of the Jc^n Test. Individual adninistration 
takes 10 to 15 minutes. 



Educational Testing 
Service 



! 

Measures ocoprehension of spoken and written Ehglish 
desigTied 'or grades 7^12. TWc fonns group adminis- 
tration takes 8S minutes. 



I 



Structure Tests ^ 
Ehglish Language (STEL) 
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Harper and Itow 



Measure knowledge of English structure and vocabulary. 
Gorelated to six placement levels with tvo forms per 
level • Group adninistration takes 30 minutes. 
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RESOURCE A: TYPES OF COST EORMOLAS 



EORMUIA 

Cost pec 
Participant 



EXAMPLE 

100 pajctlclptats anT" 
•nzollad In a pccgcam 
fundad for $100,000 
Cost! $100,000/ 
. 100/» $1,000 



RfiQUTBEHENIS 



aONSIDERATXOMS 



Cost per l^wity diCforant 

Class daasaa ara provided 

by a program funded 
at $100,000 par year 
$100,000/20 « $S,000 



Identification of total costs 
Accurate proration of Inoirect 
costs 



Established starting and ending 
datet; for classes 
Identification of tot^i costs 
Accurate proration of 
indirect costs. 



Gcxnparison of cost among programs can be 
difficult with this focnula because service^: 
offered differ In nimber and scope; clients 
nay be enrolled for different lengths of 
time among programs, and seme services cost 

more than Qthnrs, ^ 

ClassM can vary by Intensity and nianber of 
contact hours* Wot exanple, a General 
Purpose ESL class may »eet 20 hours a week 
for 24 weeks while a VESL class may meet 6 
hours a week for 12 weeks. 

A course may vary In the nunber of contact 
hours depending upon the type of student 
enrolled, roc exaapU# A General Purpose, 
iMml Om class may have 120 contact hours 
when the stutents an literate but have 300 
contact hours when the students are 
nonlifaerate. 



01 
01 

o 
o 

01 

91 

M 
O 

o 

04 



Cost per 
Contact Hour 



A progtan fianded at 
$100,000 per year 
profvldee S,000 
contact hours. 
$100,000/3000 - $20.00 
psr oontaet hour 



Cost per 
Client per 
Contact Hour 



lien students regularly 
attend a 6lass which 
costs $20.00 per 
contact hour. 
$20.00/10 • $2.00 
per client per hour 



Uhifom definition o£ contact 
hours 

Accurata calculation of contact 
hours for the year 
Identification of total costs 
Accurate proration of Indira 
costs. 



"Gontact hour"* can be defined in different 
ways^SO minutes, SS minutes, 60 minutes, etc. 

Holidays, staff vacations and unexpected 
circunstances can effect the number o£ 
contact hours. 



Average student attendance is 
used. 

Cost per contact hour is 

accurate. 

Total costs 

Identification of accurate 
proration of indirect costs. 



Some classes require a smaller enrol lii^t, 
such as literacy classes and Occupational 
S^if ic VESL. 

Classes with large enrollments may appear to 
be more cost effective, but student learning 
may be limited due to the large nunber of 
students. 

If actual attntndance per hour is used in the 
calculation instead of average daily 
ctendanoe, a lot of adninistrative tim is 
aeeded. 



er|c 
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RESOURCE ki TYPES OF COST EORMULAS 



EX)RMUIA 



EXAMPLE 



REQOIREMEirrS 



GONSIDERATIONS 



Cost per 100 cllttnta ara 

Successful enrolled in a pcogram 
Prograni which is funded at 

Gonpletion $100,000. 

SO clients ocqplete the 
program successfully. 
$100,000/50 m $2,000 



Cost per 10 students are enrolled 

Successful In a VESL class. 

Ooopietlon 8 successful ccoplete 

of a Group the class. The class 



costs $5000. 
$5,000/8 « $62S 



Standard definitions for 
program completion 
Identification of total 
costs 

Accurate proration of 
indirect costs 



for 



Standard definitions 
course canpletion. 
Identification of total 
costs 

Accurate proration of 
indirect costs 
No fault drop-outs 
defined. 



Programs may be using different definitions 
for successful conpletion. 

Programs may offer different types, levels, 
and intensity of instruction. 

Cjertain segnsnts of the client population, 
such as the nonilterate, less educated, will 
need mora secvices in order to corpleta the 
program, hence, the costs will be higher. 

Conpletion of all levels rather than job 
placement may be encouraged. 

The success of entry/exit cl asses are nore 
difficult to evaluate than the success of 
''closed'* classes. 

Ortain populations, such as nonliterate 
students, may require mora services before 
attaining success. 



Cost per 
Successful 
Achievemi»«it 
of a 

Benchmarlc 



lOO clients ara 
enrolled in a program 
which costs $100,000 
per yMT. 80 students 
gain 1 SFL In 6 months 
$50,000/80 « $£25 



Standard definitions for 

benchmarks. 

Regular assessment of 

student progress. 

Identification of total 

costs. 

Accurate proration of 
indirect costs. 
No fault drop outs 
defined. 



Recording progress will be mora ccnplex if 
class assessment is done on an ongoing 
basi^^ 

C^en entry/exit programs may measure progress 
on an ongoing basis rather than at regularly 
established points In time. 



Linda Mrowldci. Adapted from handouts for ORR Region I consultation. 
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APPENDIX G 
STEPS IN DETERMINING COS 



es 



r 

G-1 

« Steps In Deternlnlng Costs* 



Step One: 
Step Two: 

Step Three: 
Step Four: 

Step Five: 



Step Six: 



Step Seven: 



Step Eight: 



Step Nine: 



Obtain cost data from all previous vendors 
for a period of not less than two years. 

Obtain cost data from similar programs, such 
as Adult Bducatlon/BSL or Job Training 
Partnership Act (JTPA) . 

Compute as many units of cost as possible. 

Calculate the median, mean, and mode. These 
calculations Indicate clustt-s of costs. 

Examine tae high and low ranges and Identify 
possible explanations for tu*. hitfii or low 
costs. Some factors may include: 

. use of professional, full time staff 
agal«.st volunteers or part time staff. 

. higher operating costs In majo:? urban 
areas . 

. unionized salary structures In school 
districts vs. non-union community 
based agencies. 

. greater costs for Instructional supplies 
In new programs vs. continuing 
programs . 

Establish a range which appears fair and 
appropriate. 

Evaluate the applicants* costs to ensure 
that they fall within the established 
ranges. If they exceed or fall below the 
range, review the applicants' justification. 

Analyze the costs In relation to the 
specific policies for the types of 
Instruction, the priority of clients to be 
served, and the outcomes. 

Review t>e applicant's proposals to ensure 
that the applicants have the necessary 
Infrastructure resources to manage the 
progrwn and provide full and rellahlt 
accountability. 



♦Developed by Pamela I'ieubert and Lln.*^ Mrowlckl 
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